Musical Sound Information

Computer music is not the artistic expression of an exclusive set of composers that it used to be. Musicians and composers have grown to expect much more from electronics and computers than their ability to create "out-of-this-world" sounds for a tape piece. Silicon has already made its way on stage, in real-time musical environments, and computer music has evolved from being an abstract layer of sound to a substitute for real instruments and musicians. Within the past three decades, an eclectic set of tools for sound analysis and synthesis has been developed without ever leading to a general scheme which would highlight the issues, the difficulties and the justifications associated with a specific approach. A rush in the direction of incremental improvement of existing techniques has traditionally distinguished analysis and synthesis. Rather than confining ourselves to one of these arbitrarily exclusive tasks, we are pursuing an ambitious dream from a radically new perspective. The ultimate goal of this research is to infer virtual instruments from the observation of a real instrument without any strong pre-conception about the model's architecture. Ideally, the original observation should be a simple audio recording. We also want our inferred virtual instruments to exhibit physically realistic behaviors. Finally, we want the nature of the virtual instrument's control to be universal and perceptually meaningful. For this purpose, our investigation falls naturally into three steps. We first identify a set of perceptually meaningful musical gestures which can be extracted from an audio stream. In the case of a monophonic sound, we discuss the definition and the estimation of loudness, pitch contour, noisiness and brightness. The second step is to investigate means by which a physically meaningful model can be inferred from observed data. While doing so, we introduce embedding modeling as our general philosophy and reduce modeling to the characterization of prediction surfaces. We also suggest some general purpose interpretations, including an original cluster-weighted modeling technique. Finally, our third and last step is to suggest a strategy for applying such modeling ideas to musical audio streams parametrized by the perceptually meaningful musical gestures that we previously identified. We present pitch synchronous embedding synthesis (or Psymbesis), a novel approach to the inference of a virtual instrument, as a working sound synthesis algorithm and an interpretation of these suggestions. Psymbesis was designed specifically around musical instrument modeling but the general philosophy of embedding modeling extends beyond the field of computer music. For instance, we establish embedding modeling as a useful tool for the analysis of fairly small but highly non-linear deterministic dynamical systems of arbitrary nature. We expect that the introduction of embedding modeling will provide signal modeling with a new perspective, relaxing the constraint of linearity and filling the present gap between physical models and standard signal processing. Thesis Supervisor: Tod Machover Associate Professor of Music and Media Doctoral Dissertation Committee Thesis Supervisor Tod Machover Associate Professor of Music and Media Program in Media Arts and Sciences Thesis Reader Neil Gershenfeld Associate Professor of Media Arts and Sciences Program in Media Arts and Sciences Thesis Reader Rosalind Picard Associate Professor of Media Technology Program in Media Arts and Sciences Acknowledgments None of this work would have been possible without the stimulating environment of the Media Laboratory. I'd like to take this opportunity to thank my advisor, Professor Tod Machover, for inviting me to become a part of this environment back in September 1992 and for supporting my work ever since. In addition to a unique opportunity of working in a creative and artistic environment, Tod provided me with a degree of trust and confidence which has unleashed my own creativity for my research. Throughout the past four years, I've worked closely with Professor Neil Gershenfeld to whom I owe my early conversion to the philosophy of Embedding Modeling. His insights and encouragement have been an abundant source of inspiration to me in the past four years. I would also like to thank Professor Rosalind Picard for supporting my work as an insightful and detail-oriented reader. Sharing some of her ideas about modeling resulted into some major breakthroughs in the investigation of Embedding Modeling. I would probably think computer music refers to some hip techno genre if it hadn't been for Professor David Wessel who has been a mentor figure to me for the past six years. Thanks for opening my eyes and ears during the most fun internship one could ever dream of at Berkeley's Center for New Music and Audio Technology. My thanks are warmly extended to the remarkable staff and graduate student community of the Media Laboratory. Throughout four rocky years towards graduation, I was blessed by three wonderful offices-mates, Mike Wu, David Waxman, and John Underkoffler, who not only became good friends in spite of my undeniable "frenchyness", but also turned out to be among my best teachers. Rather than going through an exhaustive list of the lab's graduate students of the past four years, I'll just scream a loud thanks from the top of my lungs. I'm grateful for all of these insightful lunch breaks, trips to the trucks and to the coffee machine. I can only hope you all got as much out of them as I did. To my friends from the "good old days", Philippe, Vincent, Anne, Hervé, Suzanne, Raphaël, Benoît, Rémi, Nanou: Thank you for not holding my inexcusable silences against me. I know that materialistic details such as time and distance can't suffice to keep us apart. I'd like to thank my parents for trusting my judgment and supporting my decisions throughout the years, even when it meant an ocean between us. Finally, I would have probably never set foot anywhere close to MIT if it hadn't been for my best friend, soul-mate and wife, Karyn. For your patience, understanding and love, I'm indebted to you for life. Support for this work was provided in part by Yamaha, Sega, Motorola and TTT. Musical Sound Information – Eric Métois October 1996